Systems and methods for virtual interaction

ABSTRACT

A method for virtual interaction between a host user at a first physical location and a guest user at a second physical location is provided. The method includes receiving, by the guest assembly, environmental information relating to a host environment in the first physical location; displaying, a 3D virtual reconstruction of the host environment at the second physical location using the environmental information; and displaying a virtual avatar of the host user within the 3D virtual reconstruction. A guest user can enter a predefined area of the second physical location to be virtually transported to the host environment. The method further provides for virtual reconstruction of surfaces of the host environment and applying textures to those surfaces. A corresponding system and non-transitory computer readable medium are also described.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims the benefit of U.S. provisionalapplication No. 63/153,611 filed Feb. 25, 2021, entitled “SYSTEMS ANDMETHODS FOR VIRTUAL INTERACTION”, the entirety of which is incorporatedherein by reference.

TECHNICAL FIELD

The technical field generally relates to virtual reality (VR), augmentedreality (AR), and mixed reality (MR), and more specifically to using VR,AR and/or MR devices to facilitate virtual interaction.

BACKGROUND

Videoconferencing is a popular technology that allows users to visuallyinteract at a distance. However, this technology is not well adapted forcommunicating complex actions in 3D space. There is therefore much roomfor improvement.

SUMMARY

According to an aspect, a system for virtual interaction between a hostuser at a first physical location and a guest user at a second physicallocation is provided. The system includes: a host assembly at the firstphysical location, the host assembly including: a user interface moduleconfigured to display a virtual avatar of the guest user within a hostenvironment at the first physical location based on received interactiondata relating to the guest user, and to capture interaction datarelating to the host user; an environment detection module configured tocapture environmental information relating to the host environment; anda communications module configured to receive interaction data relatingto the guest user, and to transmit the interaction data relating to thehost user and the environmental information relating to the hostenvironment for display at the second physical location; and a guestassembly at the second physical location, the guest assembly including:a user interface module configured to: display a 3D virtualreconstruction of the host environment at the second physical locationbased on received environmental information relating to the hostenvironment; display a virtual avatar of the host user within the 3Dvirtual reconstruction of host environment at the second physicallocation based on received interaction data relating to the host user;and capture interaction data relating to the guest user within thevirtual reconstruction of the host environment; and a communicationsmodule configured to receive the interaction data relating to the hostuser, and to transmit the interaction data relating to the guest userfor displaying the virtual guest avatar within the host environment atthe first physical location.

According to an aspect, a method for virtual interaction between a hostuser at a first physical location and a guest user at a second physicallocation is provided. The method includes: receiving, via a hostassembly, interaction data relating to the guest user; displaying, basedon the received interaction data, a virtual avatar of the guest userwithin a host environment in the first physical location; capturing, viathe host assembly, interaction data relating to the host user within thehost environment; capturing, via the host assembly, environmentalinformation relating to the host environment; and transmitting theinteraction data relating to the host and the environmental informationrelating to the host environment to a guest assembly, for virtuallyreconstructing the host environment at the second physical location in3D and displaying a virtual avatar of the host user therein.

According to an aspect, a method for virtual interaction between a hostuser at a first physical location and a guest user at a second physicallocation is provided. The method includes: receiving, via a guestassembly, environmental information relating to a host environment inthe first physical location; displaying, based on the receivingenvironmental information, a 3D virtual reconstruction of the hostenvironment at the second physical location; receiving, via a guestassembly, interaction data relating to the host user; displaying, basedon the received interaction data, a virtual avatar of the host userwithin the virtual reconstruction of host environment in the secondphysical location; capturing, via the guest assembly, interaction datarelating to the guest user within the virtual reconstruction of the hostenvironment; and transmitting the interaction data relating to the guestto a host assembly, for displaying a virtual avatar of the guest userwithin the host environment at the first physical location.

According to an aspect, a system for virtual interaction between a hostuser in a host environment at a first physical location and a guest userat a second physical location is provided. The system includes a guestassembly at the second physical location, the guest assembly including:a user interface module configured to display a 3D virtualreconstruction of the host environment at the second physical locationbased on received environmental information relating to the hostenvironment, display a virtual avatar of the host user within the 3Dvirtual reconstruction of the host environment based on receivedinteraction data relating to the host user, and capture interaction datarelating to the guest user within the virtual reconstruction of the hostenvironment; and a communications module configured to receive theenvironmental information relating to the host environment, receive theinteraction data relating to the host user in the host environment, andtransmit the interaction data relating to the guest user.

According to an aspect, a method for virtual interaction between a hostuser at a first physical location and a guest user at a second physicallocation is provided. The method includes: receiving, via a guestassembly at the second physical location, environmental informationrelating to a host environment in the first physical location;displaying, by the guest assembly, a 3D virtual reconstruction of thehost environment at the second physical location using the receivedenvironmental information; receiving, via the guest assembly,interaction data relating to the host user; displaying, by the guestassembly, a virtual avatar of the host user within the 3D virtualreconstruction of host environment in the second physical location usingthe received interaction data relating to the host user; capturing, viathe guest assembly, interaction data relating to the guest user withinthe 3D virtual reconstruction of the host environment; and transmitting,by the guest assembly to a host assembly at the first physical location,the interaction data relating to the guest user, the interaction dataallowing to display a virtual avatar of the guest user within the hostenvironment at the first physical location.

According to an aspect, a non-transitory computer-readable medium isprovided. The non-transitory computer-readable medium has instructionsstored thereon which, when executed, cause a processor to carry out themethods as described above.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an exemplary embodiment of asystem for virtual interaction.

FIGS. 2A and 2B are schematics illustrating a method for virtualinteraction from the perspective of a host user, according to anembodiment.

FIGS. 3A and 3B are schematics illustrating a method for virtualinteraction from the perspective of a guest user, according to anembodiment.

FIGS. 4A and 4B are respectively a schematic and a perspective view of aportable streaming device, according to an embodiment.

DETAILED DESCRIPTION

Broadly described, the present disclosure relates to using VR, AR and/orMR devices to facilitate remote interactions in 3D space. One or moreguest users can be virtually transported into a physical environment ofa host user. The host user will be able to see guest avatars in theirphysical environment and will be able to interact with the guest usersvia their avatars. Meanwhile, the guest users will be able to see thephysical environment of the host user in 3D, while also being able tocommunicate with the host user and/or other guest users using theiravatars.

With reference to FIG. 1, an exemplary system 1 for virtual interactionis shown according to an embodiment. The system 1 includes a hostassembly 100, one or more auxiliary devices 109, and one or more guestassemblies 200 configured to interact with one another. The system canfurther include an authorization server 500 to authorize and/or brokerinteractions and sharing of data between host assembly 100, one or moreauxiliary devices 109 and one or more guest assemblies 200. The hostassembly 100 and auxiliary devices 109 are associated with a firstphysical location, whereas the guest assemblies 200 are associated withone or more second physical locations that are separate and distinctfrom the first physical location. In this configuration, the hostassembly 100 can allow a host user to virtually interact with one ormore guest users at the first physical location, while the guestassemblies 200 can allow the one or more guest users to interact withthe host users and/or other guest users at the second physicallocations. The auxiliary devices 109 can transmit data to the hostassembly 100 and/or to the guest assemblies 200 to enrich theinteractions at the first and/or second physical locations. Although inthe illustrated embodiment a single host assembly 100 is shown, it isappreciated that in other embodiments a plurality of host assemblies 100can be provided. In such embodiments, the plurality of host assemblies100 can allow one or more host users at the first physical location tointeract with one or more guest users at the second physical location.

As will be described in more detail hereinafter, the host assembly 100can be configured to carry out a number of functionalities, includingvirtually augmenting the first physical location using digital contentand allowing the host user to interact with the digital content. In someembodiments, the host assembly 100 can comprise a wearable AR/MR devicethat implements at least some of these functionalities, such as theMicrosoft HoloLens. It is appreciated, however, that other AR/MR devicesare possible as well, and that similar functionality can also beimplemented using a VR device.

In more detail now, the host assembly 100 can include a user interfacemodule 101, a communications module 103, a processing module 105, and anenvironment detection module 107. As can be appreciated, these modulesneed not be implemented in a standalone device. Instead, the hostassembly 100 can comprise one or more separate devices that includehardware and/or software to implement these modules. For example, thehost assembly 100 can comprise a wearable AR/MR device in addition toexternal sensors and an external computer or server. It should furtherbe appreciated that each module need not be implemented on a singlehardware device and/or at a single physical location. For example, somemodules can be implemented via software running on hardware at the firstphysical location, via software running on an external server (such as acloud server), or a combination of both.

The user interface module 101 can be configured to virtually project 3Ddigital content in the first physical location and receive input toallow the host user to interact with the 3D digital content. Forexample, the user interface module 101 can include a wearable displayconfigured to present stereoscopic images to the wearer. It isappreciated, however, that other types of displays are possible. Theuser interface module 101 can further include any suitable user inputdevices. Such input devices can, for example, include one or moresensors for tracking the host user's body movements (including headmovements and/or hand gestures) through preferably 6 degrees of freedom,one or more sensors for tracking the host user's position in thephysical location, one or more sensors for capturing the host user'svoice, one or more handheld controllers, etc. As can be appreciated,some of these sensors can be integrated in the wearable display whileothers can be separate hardware devices, such as smart cameraspositioned throughout the first physical location.

The communications module 103 can be configured to send and/or receivedata from external devices via different communication channels and/ordifferent protocols. In the present embodiment, the communicationsmodule 103 allows host assembly 100 to communicate with one or moreguest assemblies 200 via a network 3. The communications can, forexample, be facilitated and/or brokered via authorization server 500that can also be reached via network 3. The data exchanged over network3 can allow users to interact with one another using their respectivehost 100 or guest 200 assemblies. As can be appreciated, different typesof networks are possible, such as a personal area network (PAN), a localarea network (LAN), a wide area network (WAN), among others. Dependingon the type and extent of network 3, users can interact with one anotherfrom greater distances. For example, if network 3 corresponds to a LAN,users can interact between physical locations corresponding to differentspaces of a common building or campus. If the network 3 corresponds to aWAN, such as the internet, interaction between more distant physicallocations is possible, such as spaces in different cities or countries.

In the present embodiment, the communications module 103 is alsoconfigured to send and/or receive data from one or more auxiliarydevices 109 associated with the first physical location. Auxiliarydevices 109 can be any type of device that can provide information thatcan be used to enhance the 3D digital content displayed to the host useror guest users, and/or that can assist in recreating environmentalelements that are present in the first physical location. Thecommunications module 103 can be configured to interface with theauxiliary devices 109 to receive data therefrom, for example in the formof video, audio or data streams, among others. As an example, in anembodiment where system 1 is used in a medical setting, auxiliary device109 can comprise a patient monitoring system 120 having a display. Thecommunications module 103 can be configured to interface with auxiliarydevice 109 and receive data from the patient monitoring system 120, suchas patient vital information recorded by the patient monitoring system120 and/or data that reflects what is shown on the display of thepatient monitoring system 120 at any given time. Although a patientmonitoring system 120 is described, it is appreciated that auxiliarydevice 109 can comprise other devices that are associated with the firstphysical location.

In some embodiments, the auxiliary device 109 can comprise a hardwareinterface acting as a bridge to facilitate sending data to and/orreceiving data from one or more devices associated with the firstphysical location. By way of example, the auxiliary device 109 cancomprise a portable streaming device 150. The portable streaming device150 can be configured to interface with a device associated with thefirst physical location, such as a patient monitoring device 120 in theillustrated embodiment. The portable streaming device 150 can capturedata from device 120, and transmit the captured data remotely, forexample to host assembly 100 via communications module 103 and/or toauthorized guest assemblies 200. In the present embodiment, the network3 corresponds to the internet, and the streaming device 150 isconfigured to capture a video signal from device 120 and to stream thecaptured video securely over the internet to authorized users. Morespecifically, the streaming device 150 has a dedicated connection to theinternet through which peer-to-peer communication with other devices canbe established, and through which the video stream can be transmitted.It is appreciated, however, that other configurations are possible. Forexample, in some embodiments, the streaming device 150 can interfacedirectly with host assembly 100 via communications module 103, totransmit the video stream directly and/or over a local network insteadof over the internet. In some embodiments, the streaming device 150 caninterface directly with host assembly 100 via communication module, andutilize the host assembly's connection to the internet to establishpeer-to-peer communications with other devices and/or to transmit thevideo stream to other devices. In some embodiments, the streaming device150 can be configured to transmit other types of data acquired fromdevice 120, such as an audio stream, a data stream, etc.

In more detail now, and with reference to FIGS. 4A and 4B, an exemplaryportable streaming device 150 is shown according to an embodiment. Thestreaming device 150 comprises a housing 151 made from aluminum. It isappreciated, however, that the housing can be made of other materialsthat facilitate heat dissipation and/or that are suitable for use inmedical environments. The housing supports a processing module 153operatively connected to an input module 155, an output module 157, anda user interface module 159. In the present embodiment, the processingmodule 153 comprises a mini-PC configured to operate in a kiosk mode andconfigured to automatically run an application at start-up to enablecapture and streaming of a video signal via input 155, output 157 anduser interface 159 modules. It is appreciated, however, that otherprocessing hardware is also possible. It is further appreciated that thestreaming device 150 and its modules can be powered via any suitablemeans. For example, in the present embodiment, an external power supplycan be connected via power connector 152. In other embodiments, thepower supply can be at least partially integrated in the housing 151,for example in the form of a battery.

The input module 155 is configured to interface with an external device120 and receive data therefrom. In the present embodiment, the inputmodule 155 comprises an input HDMI port 156 configured to receive avideo signal from an HDMI output of device 120. It is appreciated thatother ports and/or connections are possible depending on the data to bereceived from the external device 120 and/or the types of connectionsavailable on the external device 120. For example, in some embodiments,the input HDMI port 156 can be configured to receive an audio signal. Insome embodiments, different wired or wireless connection types can beused to interface with external device 120 and receive data therefrom.The input module 155 can further be configured to convert an inputsignal from external device 120 into a format suitable for theprocessing module 153. For example, in the present embodiment, the inputmodule 155 is configured to convert an audio/video HDMI signal receivedvia HDMI port into a USB signal for input to the mini-PC in processingmodule 153.

The output module 157 is configured to enable communications withexternal devices, for example to establish connections with authorizeddevices over a network and share a real-time video stream. The outputmodule 157 can comprise a wireless radio and corresponding antenna 158,allowing the streaming device 150 to wirelessly connect to a network. Inthe present embodiment, the wireless radio corresponds to a Long-TermEvolution (LTE) radio, providing the streaming device 150 with a directbroadband connection to the internet. It is appreciated that other wiredand/or wireless connections are possible to allow the streaming device150 to communicate over the internet, over a local network, and/ordirectly with host assembly 100. In some embodiments, the connection canprovide a bandwidth of at least 2 Mbps to allow real-time streaming of1080P video. The output module 157 can further be configured to converta video/audio signal into a format suitable for secure remote streaming,for example by compressing the video/audio signal, encrypting thevideo/audio signal, and/or separating the signal into packets forreal-time transmission over a network.

In some embodiments, the output module 157 can include an outputinterface, such as an output HDMI port, for repeating video signals orother data received via input module 155. As can be appreciated, theoutput interface can allow chaining multiple streaming devices 150together to provide redundancy for streaming in case one streamingdevice fails 150. For example, a first streaming device can interfacewith external device 120 via input module 155 of first streaming device,and with a second streaming device via output module 157 of firststreaming device. A signal received via input module 155 can be streamedremotely by output module 157 via a wireless radio while also beingpassed along to the second streaming device by the output module viaoutput interface. The second streaming device can receive the signalfrom first streaming device via its input module 155 and stream remotelyby its output module 157 on a separate, parallel stream. Subsequentstreaming devices can also be chained to second streaming device in asimilar manner if needed. It is appreciated, however, that otherconfigurations are possible for providing redundancy. For example, aplurality of streaming devices can be connected to external device 120in parallel (such as via a signal splitter) instead of being chained inseries.

The user interface module 159 is configured to allow a user to interactwith the streaming device 151 and confirm proper operation. In thepresent embodiment, the user interface module 159 comprises an LCDscreen 160 allowing a user to have a live view of the video streamreceived via the HDMI port 156. The screen 160 can further be providedwith touch functionality, for example to receive user inputs forcontrolling applications running on the processing module 153. Anexternal power button 161 is also provided to allow a user to easilypower the device on or off as needed. The processing module 153 can beconfigured to run a software application that facilitates streaming ofdata received via input module 155 to authorized devices via outputmodule 157. The application can be configured to run automatically whenthe streaming device 150 is powered on. When running, the applicationcan cause the streaming device 150 to communicate with authorizationserver 500 to initiate a stream. More specifically, the streaming device150 can create a session with the server 500 and send information to theserver 500 indicating that it is ready to stream. When an externaldevice, such as a host assembly 100 and/or one or more guest assemblies200, wishes to receive the stream, the external device can firstcommunicate with the authorization server 500. The authorization server500 can determine whether the user of the external device is authorized(for example using any suitable authentication service, such as AzureActive Directory), and if the user is authorized, the server 500 canexchange information (such as connection parameters) between thestreaming device 150 and the external device such that they canestablish a direct peer-to-peer connection over which the stream can betransmitted. The streaming device 150 can be configured to encrypt thestream prior to transmittal (for example via AES-128) which can then bedecrypted once received by the external device. Accordingly, theinformation exchanged by server 500 can include encryption/decryptionkeys to enable end-to-end encryption between the streaming device 150and the external devices receiving the streams.

In some embodiments, one or more streaming devices 150 can have theirstreams grouped into virtual rooms by the authorization server 500. Forexample, streams from a plurality of streaming devices 150 associatedwith the same physical location can be grouped into a same virtual room,whereas streaming devices 159 associated with a different physicallocation can be grouped into a different room. In such embodiments, auser authorized to access a given room can be provided access to allstreams contained within those rooms. When powering on the streamingdevice 150, a prompt can be presented via user interface module 159, forexample on touchscreen 160, prompting a user to select an existing roomor to create a new room. The user's selection can subsequently becommunicated to server 500 to place the stream from the streaming device150 in the appropriate virtual room and manage permissions accordingly.

Referring back to FIG. 1, the processing module 105 of host assembly 100can be operatively coupled to the user interface module 101,communication module 103 and environment detection module 107, and canbe configured to carry out any processing required by such modules. Asan example, processing module 105 can be configured to generate and/ororganize 3D digital content for display to the host user, to interpretand/or act responsive to data received from user input devices,auxiliary devices 109, and guest assemblies 200, etc. In someembodiments, processing module 105 can comprise memory storingcomputer-readable program instructions, and a processor for executingsaid instructions. It is appreciated, however, that other configurationsare possible. For example, processing module 105 can comprise anapplication-specific integrated circuit (ASIC) and/or any other hardwarecapable of carrying out logical operations. It is further appreciatedthat processing module 105 can carry out some processing tasks on adevice worn by the host user, and other processing tasks on a separatecomputing device such as a computer or remote server (such as a cloudserver).

Environment detection module 107 can be configured to detect, map and/orvirtually model an environment of the first physical location, includingenvironmental elements such as surfaces (including ceiling and walls)and physical objects positioned throughout the environment. The module107 can comprise any suitable sensors capable of detecting objects inthe physical location, such as one or more cameras or other scanningdevices. In some embodiments, the module 107 can comprise sharedsensors, for example utilizing one or more cameras integrated in awearable device for both detecting host user's inputs/gestures andmodelling the environment of the first physical location. In someembodiments, the module 107 can comprise dedicated sensors, such as aseries of fixed cameras positioned throughout the first physicallocation. Additional dedicated sensors can be provided in areas thatrequire more precise monitoring. For example, a dedicated camera can beprovided to closely follow an object of interest.

As can be appreciated, the environment detection module 107 can beconfigured to capture and segment volumetric data of the environment inorder to generate 3D models of different physical objects in the firstphysical location, for example using a volumetric sensor to generatepoint clouds or other 3D modelling techniques. The module 107 can beconfigured to identify objects and track their position and/or movementsin the first physical location. In some embodiments, the module 107 canfurther be configured to sample object surfaces, for example bycapturing 2D images thereof. In some embodiments, dedicated sensors canbe provided to sample object surfaces. For example, a dedicated cameracan be provided to capture detailed images and/or continuous video of adisplay that is part of an object in the first physical location (suchas a display of a patient monitoring device). The environment detectionmodule 107 can further include sensors for detecting position and/ororientation of a camera used to capture images in the environment.

As will be described in more detail hereinafter, one or more guestassemblies 200 can be provided for interacting with the host user. Eachguest assembly 200 can be configured to virtually recreate theenvironment of the host user so that the guest user can interacttherein. Recreating the environment can comprise virtuallyreconstructing physical objects in addition to any digital content thathas been added to the environment. As can be appreciated, the guestassembly 200 can comprise any suitable hardware for immersing a guestuser in a virtual 3D environment. For example, the guest assembly 200can comprise a wearable AR device. It is appreciated, however thatsimilar functionalities can also be carried out using AR/MR devices.

In more detail now, the guest assembly 200 can include a user interfacemodule 201, a communications module 203, a processing module 205, and anenvironment detection module 207. As can be appreciated, these modulesneed not be implemented in a standalone device. Instead, the guestassembly 200 can comprise one or more separate devices that includehardware and/or software to implement these modules. For example, theguest assembly 200 can comprise a wearable VR device in addition toexternal sensors and an external computer or server. It should furtherbe appreciated that each module need not be implemented on a singlehardware device and/or at a single physical location. For example, somemodules can be implemented via software running on hardware at the firstphysical location, via software running on an external server (such as acloud server), or a combination of both.

The user interface module 201 can be configured to immerse a guest userin a virtual 3D environment and receive input to allow the guest user tointeract with the 3D digital content in the environment. For example,the user interface module 201 can include a wearable display configuredto present stereoscopic images to the wearer as the guest user movesthrough a second physical location. It is appreciated, however, thatother types of displays are possible. The user interface module 201 canfurther include any suitable user input devices. Such input devices can,for example, include one or more sensors for tracking the guest user'sbody movements (including head movements and/or hand gestures) throughpreferably 6 degrees of freedom, one or more sensors for tracking theguest user's position in the physical location, one or more sensors forcapturing the guest user's voice, one or more handheld controllers, etc.As can be appreciated, some of these sensors can be integrated in thewearable display while others can be separate hardware devices, such assmart cameras positioned in the second physical location.

The communications module 203 can be configured to send and/or receivedata from external devices via different communication channels and/ordifferent protocols. In the present embodiment, the communicationsmodule 203 allows guest assembly 200 to communicate with host assembly100 via the network 3, as described above. The communications module 203can further be configured to communicate with auxiliary device 109 overnetwork 3, for example to receive video, audio or other data streamsfrom the auxiliary device 109. The processing module 205 can beoperatively coupled to the user interface module 201, communicationmodule 203, and environment detection module 207, and can be configuredto carry out any processing required by such modules. As an example,processing module 205 can be configured to generate and/or organize 3Ddigital content for display to the guest user, to interpret and/or actresponsive to data received from user input devices and host assembly100, etc. In some embodiments, processing module 205 can comprise memorystoring computer-readable program instructions, and a processor forexecuting said instructions. It is appreciated, however, that otherconfigurations are possible. For example, processing module 205 cancomprise an application-specific integrated circuit (ASIC) and/or anyother hardware capable of carrying out logical operations. It is furtherappreciated that processing module 205 can carry out some processingtasks on a device worn by the guest user, and other processing tasks ona separate computing device such as a computer or remote server (such asa cloud server).

Environment detection module 207 can be configured to detect, map and/orvirtually model an environment of the second physical location. Themodule 207 can comprise any suitable sensors capable of detecting theenvironment of the second physical location and/or objects containedtherein, such as one or more cameras or other scanning devices. As canbe appreciated, the environment detection module 207 can allow providinga guest user with feedback about their immediate physical environment,particularly where the guest user's view of the physical environmentwould otherwise be obstructed (for example while wearing an opaque VRheadset as part of user interface module 201). In some configurations,the environment detection module 207 can include a camera that allowsproviding a stereoscopic view of the guest user's immediate physicalenvironment while wearing a VR headset, effectively recreating what theguest user would see when not wearing the VR headset. This can allow theguest user to view their environs without removing the VR headset. Inother configurations, the environment detection module 207 can include aproximity sensor which can, for example, be used to warn the guest userif they are approaching an object or wall in their physical environmentto avoid physical collisions while the guest user is completely immersedin a virtual environment while wearing VR headset.

As mentioned above, the system 1 can allow one or more host users tovirtually interact with one or more guest users, while virtuallytransporting the guest users into an environment of the host user orusers. With additional reference to FIGS. 2A and 2B, an exemplary methodfor virtual interaction will be explained from the perspective of a hostuser. As shown in FIG. 2A, the host user 301 is situated in a hostenvironment 300. The host environment 300 can correspond to any 3D spaceat a first physical location where the host user 301 is free to explore.In some embodiments, the space can be delimited by physical barriers,such as walls. In further embodiments, the space can be virtuallydelimited. For example, the host environment 300 can correspond to adefined area of a larger space, such as a predetermined area that isequipped for tracking movements and interactions of the host user 301therein. In some embodiments, the host environment 300 can be delimitedby virtual barriers, such as by virtual walls 311 that can be hidden orvirtually projected within the host environment 300 and visible to hostuser 301.

The host environment 300 includes environmental elements such asphysical objects with which host user 301 can physically interact. Inthe present embodiment, the environmental elements include a table 305,and a patient monitoring system 307 which includes a display 309. It isappreciated, however, that other environmental elements are possible. Inthe present embodiment, the environmental elements 305, 307 are mobilein that the host user 301 can move them freely throughout the hostenvironment 300 via physical manipulation. It is appreciated, however,that some environmental elements can be fixed.

As shown in FIG. 2B, the host user 301 interacts with guest user 401 inthe host environment 300 via host assembly 100. More specifically, inthe illustrated embodiment, the host user 301 wears an AR/MR headset 303that virtually projects 3D digital content into the host environment300. While wearing the AR/MR headset 303, the host user 301 can continueto see the host environment 300 and environmental elements therein,while also being able to see spatially aware 3D digital contentprojected throughout the environment 300. The 3D digital contentincludes at least an avatar 401′ corresponding to guest user 401,thereby allowing the host user 301 to see and interact with a virtualrepresentation of the guest user 401. The guest avatar 401′ iscontrolled by the guest user 401 and reflects the currentposition/perspective of the guest user in the 3D space of the hostenvironment 300 as the guest user explores a virtual representation ofthe host environment. In some embodiments, the guest avatar 401′ canreflect gestures that are made by the guest user 401. As can beappreciated, in some embodiments, a plurality of host users can eachwear a respective AR/MR headset such that each host user can bepresented with the spatially aware 3D digital content projected intohost environment 300.

In the present embodiment, the 3D digital content also includes at leastone virtual object 310. The virtual object 310 can correspond to any 2Dor 3D model that can be virtually projected in the host environment 300to facilitate communication between host user 301 and guest user 401. Insome implementations, virtual object 310 can be configured to resemblefamiliar physical objects, such as medical tools or devices. In someimplementations, virtual object 301 can be configured to presentinformation such as an image, a video, a document (such as a pdf), awebpage or other graphical user interface for software, etc. In thepresent embodiment, the virtual object 310 is a shared object in that itis visible to both the host user 301 and guest user 401. It isappreciated, however, that some virtual objects can be private virtualobjects, and thus be visible only to host user 301 or only to guest user401.

In some embodiments, the virtual object 310 can comprise a video streamreceived from auxiliary device 109. Live video received from theauxiliary device 109 can thus be displayed as part of a 2D or 3D sharedor private virtual object 310 positioned within the host environment300. For example, the video received from auxiliary device 109 cancorrespond to a live view of display 309 of the patient monitoringdevice 307. A virtual object 310 comprising the video can be positionedwithin the virtual environment, such that a live virtual copy of display309 is provided at a convenient location. This can be useful, forexample, if the actual display 309 is not at a convenient location,and/or if it would be inconvenient to physically reposition the patientmonitoring device 307 and/or its display 309. It can also be useful inproviding a guest user 401 with a dedicated and/or clearer view of whatis being shown on the display 309.

In the illustrated embodiment, the shared virtual object 310 can befreely manipulated and positioned in 3D space by both the host user 301and guest user 401. It is appreciated that in some embodiments,permissions to manipulate and position the shared virtual object 310 canbe restricted to only the host user 301 or guest user 401. It is furtherappreciated that if the virtual object 310 corresponds to a privateobject, the virtual object 310 can be positioned and manipulated only bythe user to which it belongs.

In some embodiments, the virtual object 310 can be positioned at anabsolute position within host environment 300 such that once positioned,the virtual object 310 can remain at the same 3D position in the hostenvironment 300 until it is repositioned. It is appreciated, however,that in some embodiments, virtual objects 310 can be positioned atrelative positions within the host environment 300.

As an example, virtual objects 310 can be positioned relative tophysical objects in the host environment, such that the virtual objects310 follow movement of the physical objects as they move through 3Dspace. As an example, virtual object 310 can correspond to a virtualsticky note that can be attached to patient monitoring system 307, andthat can follow patient monitoring system 307 as the monitoring systemmoves about the host environment 300. This can be implemented by hostassembly 100 and/or guest assembly 200 using any suitable process. Forexample, the process can involve determining a position of a physicalobject in the host environment 300, determining a position of a virtualobject relative to the physical object, identifying a movement of thephysical object and, in response thereto, determining a new position ofthe physical object, determining a new position of the virtual object tomaintain the same relative position to the physical object in the newposition of the physical object, and repositioning the virtual objectinto the determined new position. As can be appreciated, the process canbe carried out continuously such that the virtual object can follow thephysical object in real time while the physical object is being moved.

As another example, virtual objects 310 can be positioned relative tohost user 301, guest user 401 and/or their avatars 301′, 401′, such thatthe virtual objects follow movement of the host user 301, guest user 401and/or their avatars 301′, 401′ as they move through 3D space. Thevirtual object can be positioned relative to the host user 301 and/orhost avatar 301′, for example at a fixed distance relative thereto, suchas at an arm's length. As the host user 301 and/or host avatar 301′moves about in 3D space, the virtual object can follow such that isalways remains at arm's length until repositioned to another relative orabsolute position. As can be appreciated, where virtual object comprisesa video received from auxiliary device, the video from the auxiliarydevice can always remain at arms length to the host user 301 and/orguest user 401. The host user 301 and/or guest user 401 need only orienttheir field of view in the direction of the positioned virtual object tosee the video from the auxiliary device. As can be appreciated, this canbe implemented by host assembly 100 and/or guest assembly 200 using anysuitable process. For example, the process can involve determining aposition of host user 301 in the host environment 300, determining aposition of a virtual object relative to host user 301, identifying amovement of the host user 301 and, in response thereto, determining anew position of the host user 301, determining a new position of thevirtual object to maintain the same relative position to the host user301, and repositioning the virtual object into the determined newposition. As can be appreciated, the process can be carried outcontinuously such that the virtual object can follow the host user 301in real time while the host user 301 moves about. It is appreciated thata similar process can be carried out to cause a virtual object to followhost avatar 301′, guest user 401 and/or guest avatar 401′.

As yet a further example, virtual objects 310 can be positioned relativeto a field of view of the host user 301 or guest user, such that thevirtual objects can remain in a fixed position in the host or guestuser's 301, 401, field of view as they move about and/or change theirfield of view in 3D space. As an example, the virtual object can bepositioned such that it always remains in a top-right corner of the hostuser's field of view. Where the virtual object comprises a videoreceived from auxiliary device, the video from the auxiliary device willalways be visible to host user regardless of their current position orfield of view in the host environment. As can be appreciated, this canbe implemented by host assembly 100 and/or guest assembly 200 using anysuitable process. For example, the process can involve determining aposition of host user 301 in the host environment 300, determine a fieldof view of the host user 301 in the host environment, determining aposition of a virtual object relative to host user's 301 field of view,identifying a movement of the host user 301 and/or a change of the hostuser's 301 field of view and, in response thereto, determining a newposition and field of view of the host user 301, determining a newposition of the virtual object to maintain the same relative position tothe host user's 301 field of view, and repositioning the virtual objectinto the determined new position. As can be appreciated, the process canbe carried out continuously such that the virtual object can follow thehost user 301 in real time while the host user 301 moves about. It isappreciated that a similar process can be carried out to cause a virtualobject to follow host avatar 301′, guest user 401 and/or guest avatar401′.

In some embodiments, tracking of physical objects in the hostenvironment 300 can be carried out by the host assembly 100. The hostassembly 100 can subsequently transmit to the guest assembly 200information relating to new positions of physical objects and/or virtualobjects such that the guest assembly 200 can display the physical and/orvirtual objects to guest user 401 at their correct positions in realtime. In some embodiments, the host assembly 100 can gather and transmitfurther information about the host environment 300 and physical objectstherein to facilitate interaction and assist in virtually reconstructingthe host environment 300 at another physical location. For example, thehost assembly 100 can be configured to gather volumetric informationrelating to physical objects in the host environment 300, such as bymeasuring and/or generating 3D models of physical objects viaenvironment detection module 107, and transmitting such information tothe guest assembly 200. In some embodiments, generating 3D models cancomprise processing raw scanning data, for example by smoothing scannedsurfaces to generate volumetric models without holes or discontinuities.In some embodiments, a reflective solution can be applied to surfaces ofphysical objects in the host environment 300 to facilitate scanning. Thehost assembly 100 can further be configured to gather surfaceinformation relating to physical objects, such as by sampling surfacecolors and/or capturing surface texture of physical objects andtransmitting such information to the guest assembly 200. In someembodiments, the sampled colors and/or textures can be used to recognizepredefined materials or patterns, and information relating to thepredefined materials or patterns (such as an identifier) can becommunicated to the guest assembly 200.

In an embodiment, the host assembly 100 can comprise a HoloLens or othersimilar AR/MR device that can be used to gather and transmit informationabout the host environment that can subsequently be used to virtuallyreconstruct the host environment 300 remotely. A process ofreconstructing the host environment can begin with a volumetric scan ofthe host environment. This can include scanning surfaces in the hostenvironment via environment detection module 107, to construct a 3Dpolygonal mesh thereof. The scan can, for example, be carried out usingdepth sensors provided by the HoloLens and/or other sensors or cameraspositioned within the host environment. The scan can be conducted inreal time and in different qualities as needed to produce a surface meshhaving different polygon sizes. The results of the scan carried out bythe HoloLens can be transmitted remotely, for example as a binary fileto a remote server and/or directly to guest assembly 200. The scan canbe refreshed regularly as needed, for example at intervals of about 10seconds. In the present embodiment, each time the scan is refreshed, thefull binary file containing the complete scan of the host environment300 can be transmitted. It is appreciated, however, that otherconfigurations are possible. For example, in some embodiments, once aninitial scan is sent, changes to the scan can be identified duringsubsequent refreshes, and only changes to the scan need be transmitted.In some embodiments, a plurality of scans can be conducted in parallel,such as where a plurality of host assemblies and/or AR/MR devices areprovided. The plurality of scans can be transmitted separately and/orcan be combined into a single binary file.

The mesh from the volumetric scan can be used to reconstruct surfacesdetected in the host environment 300. As can be appreciated, the meshcontains volumetric information only without surface texture.Accordingly, subsequent steps can be carried out to capture textureinformation that can be applied to the mesh to more faithfullyreconstruct the appearance of the surfaces detected in the hostenvironment 300.

In an embodiment, texture information can be acquired by capturing colorimages of the host environment 300. The images can be captured viaenvironment detection module 107 of the one or more host assemblies, forexample using a front facing RGB camera provided by the HoloLens and/orother cameras positioned within the host environment. In the presentembodiment, the images are captured in the form of a video, for examplehaving a resolution in a range of about 896×504 pixels up to about2272×1278 pixels, but it is appreciated that other configurations arepossible. For example, in some embodiments, high-resolution images ofthe host environment can be captured at regular intervals.

Acquiring texture information can further include capturing perspectiveinformation along with the images of the host environment 300. Morespecifically, the perspective information can include, for each capturedimage, a position and orientation of the camera within the hostenvironment 300 at the moment the image was captured (ex: x, y, zposition, pitch, yaw, roll orientation). As can be appreciated, theperspective information can be used to properly position and transformthe image for projection on the appropriate surface when reconstructingthe host environment.

Once acquired, the texture information can be transmitted remotely, forexample to a server and/or directly to guest assembly 200. In someembodiments, the texture information can be transmitted in real time,for example as one or more streams over a peer-to-peer connection withguest assembly 200. The texture information can include a video streamcomprising images of the host environment synchronized with acorresponding stream of perspective information. In some embodiments,the video and perspective information streams can be transmittedseparately and/or in parallel, while in other embodiments theperspective information can be embedded in the video stream and/or ineach image. As an example, each captured image in the video stream canbe encoded to include its corresponding perspective information, such asby embedding data in the first two rows of pixels in the image.

In some embodiments, salient objects can be identified from thevolumetric scan and/or from the texture information, and differentscanning techniques can be applied based on the identified object. Forexample, an object of interest can be identified using any suitableobject recognition technique, such as using artificial intelligence, anda more detailed and/or more regular volumetric scan or textureinformation acquisition can be conducted for that object. An object oflesser interest can be identified in a similar fashion, and a lessdetailed and/or less regular volumetric scan or texture informationacquisition thereof can be conducted, or the object can be omitted fromthe volumetric scan or texture information acquisition. As an example,identified moving objects can have their volumetric informationrefreshed at more regular intervals and/or be scanned or imaged lowerquality, while static objects identified in the scan can have theirvolumetric information refreshed at less regular intervals and/or bescanned or imaged at higher quality. In some embodiments, the volumetricscan and/or texture information can include only static objects, whileidentified moving objects can be omitted. In yet further embodiments,different scanning or modelling techniques can be applied to certainidentified objects. For example, if the identified object corresponds toa human, the object can be omitted from the volumetric scan and insteada skeleton of the human can be calculated and transmitted for subsequentreconstruction as an avatar instead of being included as part of thevolumetric surface mesh.

As can be appreciated, by communicating volumetric and texturalinformation, the physical objects can be faithfully reconstructedvirtually at another physical location by the guest assembly 200. Insome embodiments, raw and/or minimally processed volumetric and texturalinformation can be transmitted by the host assembly 100 to the guestassembly 200. It is appreciated, however, that other methods arepossible for communicating information about physical objects in thehost environment 300. For example, in some embodiments, the volumetricand/or surface information can be used to recognize predefined objects,and information relating to recognized objects can be communicated tothe guest assembly 200. In further embodiments, the host assembly 100can be configured to transmit to the guest assembly 200 data receivedfrom auxiliary devices. Such data can, for example, include informationthat can be used to replicate a graphical user interface of the display309 of patient monitoring system 307 or other physical object.

Turning now to FIGS. 3A and 3B, the exemplary method for virtualinteraction will be explained from the perspective of a guest user. Asshown in FIG. 3A, a guest user 401 is situated in a guest environment400. The guest environment can correspond to any 3D space at a secondphysical location where the guest user is free to explore. In someembodiments, the space can be delimited by physical barriers, such aswalls. In further embodiments, the space can be virtually delimited. Forexample, the guest environment 400 can correspond to a defined area of alarger space, such as a predetermined area that is equipped for trackingmovements and interactions of the guest user 401 therein. In anembodiment, the guest environment 400 is at least the same size as hostenvironment 300, such that there is sufficient space to virtuallyreconstruct the entirety of the host environment 300 within the guestenvironment. It is appreciated, however, that in some embodiments, theguest environment 400 can be smaller than the host environment 400. Insuch an embodiment, the virtual representation of the host environmentcan be scaled to fit within the guest environment 400 and/or only aportion of the host environment 300 can be recreated in the guestenvironment at a given time. Additionally, or alternatively, the guestuser 401 can be permitted to physically move around in a limitedpredetermined area of the host environment 300 that corresponds to thephysical space available in the guest environment 400. In someembodiments, the guest user 401 can change or select the limited area ofthe host environment 300 in which the guest user 401 can move around,for example using a controller or other input mechanism.

In the illustrated embodiment, the guest environment 400 is devoid ofphysical objects. In this configuration, the guest user 401 can movefreely throughout the guest environment 400 without physicalobstructions. This configuration can also allow for the host environment300 and environmental objects therein to be virtually recreated in theguest environment 400 without risk of occluding with objects in theguest environment 400. Although not illustrated, it will be appreciatedthat if the guest environment 400 corresponds to a define area of alarger space at the second physical location, there may exist physicalobjects outside the defined area.

As shown in FIG. 3B, the guest user 401 interacts with the host user 301in the guest environment 400 via a guest assembly 200. Morespecifically, in the illustrated embodiment, the guest user 401 wears aVR headset 403 that virtually transports and immerses the guest user 401into the host environment 300 by creating a virtual reconstruction ofthe host environment 300′ within a 3D space covered by guest environment400. In the present embodiment, while wearing the VR headset 403, theguest user's 401 view of guest environment is obstructed and replacedwith a view of a digitally reconstructed version of the host environment300′. It is appreciated, however, that other configurations arepossible. For example, in some embodiments the headset 403 cancorrespond to an AR/MR headset which overlays the 3D reconstructions ofhost environment 300′ on top of the guest environment 400, such that theguest user 401 can retain at least partial visibility on the guestenvironment 400. In the present embodiment, virtual walls 411 areprojected as part of the reconstructed host environment 300′,representing a boundary within which the host environment 300′ isreconstructed within the physical space of the guest environment 400.

In some embodiments, the VR headset 403 includes a camera that capturesimages and/or video from the perspective of the guest user 401. In suchan embodiment, the guest user can be provided with a virtual view ofguest environment 400 as captured by the camera, even though the guestuser's 401 actual view of the guest environment 400 is obstructed by theVR headset 403. In some configurations, the guest user 401 can manuallyswitch between the reconstructed view of the host environment 300′ andthe virtual view of the guest environment 400. In some configurations,the view presented to the guest user can be based on a determination ofthe guest user's position within the guest environment 400. As anexample, if it is determined that the guest user is positioned withinthe boundary defined by virtual walls 411, the reconstructed view of thehost environment 300′ can be displayed to the guest user 401 via VRheadset 403. Similarly, if it is determined that the guest user ispositioned outside the boundary defined by virtual walls 411, thevirtual view of the guest environment 400 can be displayed to the guestuser 401 via VR headset 403.

When the guest user 401 is positioned within the boundary defined byvirtual walls 411 and the reconstructed view of the host environment300′ is displayed, the virtual walls 411 can be projected as part of theview to indicate the boundaries of the reconstructed host environment300′. In some embodiments, the virtual walls 411 can be opaque whenviewed from within the boundaries of the reconstructed host environment300′ such that the guest user 401 cannot see past the virtual walls 411and outside the reconstructed host environment 300′. In otherembodiments, the virtual walls 411 can be transparent or semitransparentwhen viewed from within the boundaries of the reconstructed hostenvironment 300′, such that when looking in the direction of virtualwalls 411, the guest user 401 can see past the virtual walls 411 and beprovided with the virtual view of the guest environment 400 beyond theboundaries of the reconstructed host environment 300′. Similarly, whenthe guest user 401 is positioned outside the boundary defined by thevirtual walls 411 and the virtual view of the host environment 400 isdisplayed, the virtual walls 411 can be projected to indicate theboundary within which the reconstructed host environment 300′ islocated. In some embodiments, the virtual walls 411 can be opaque whenviewed from outside boundaries of the reconstructed host environment300′, such that the guest user 401 cannot see past the virtual walls 411and into the reconstructed host environment 300′. In other embodimentsthe virtual walls 411 can be transparent or semi-transparent when visedfrom outside the boundaries of the reconstructed host environment 300′,such that when looking in the direction of virtual walls 411, the guestuser 401 can see past the virtual walls 411 and be provided with a viewof the reconstructed host environment 300′ within the boundaries definedby the virtual walls 411. In some embodiments, the virtual walls 411 canbe transparent or semi-transparent when viewed from outside, whileopaque when viewed from inside, and vice-versa. In some embodiments, thedisplay of the reconstructed host environment 300′ and/or the virtualview of the guest environment 400 can be visually altered to facilitatevisually distinguishing between views of the reconstructed hostenvironment 300′ and the guest environment 400. For example, thereconstructed host environment 300′ can be displayed in color, while thevirtual view of the guest environment 400 can be displayed in grayscale.

In the present embodiment, the guest user 401 can move around thevirtually reconstructed host environment 300′ by moving around in thephysical space of the guest environment 400. More specifically, as theguest user 401 moves about in the guest environment 400, the 3D viewpresented to the guest user 401 is updated such that the guest user'sfield of view corresponds to what the guest user 401 would perceive ifthey were physically present at the same relative position in the hostenvironment 300. It is appreciated, however, that other configurationsare possible for allowing the guest user 401 to explore the 3Dreconstruction of the host environment 300′. For example, in someembodiments, the guest assembly 200 can include a control mechanism(such as controller with a joystick, point-and-click gesturing, etc.)that allows the guest user 401 to change their perspective or viewwithin the reconstructed host environment 300′ without physicallydisplacing in the guest environment 400. In other words, the controlmechanism can allow the guest user 401 to virtually teleport todifferent locations within the reconstructed host environment 300′. Insome embodiments, the guest user 401 can explore the reconstructed hostenvironment 300′ using a combination of the above. For example, inembodiments where physical limitations of the guest environment 400would prevent the guest user 401 from physically exploring the entiretyof the reconstructed host environment 300′ (for example when the guestenvironment 400 is smaller than the host environment 300), the guestuser 401 can physically explore a limited predefined area of thereconstructed host environment 300′ by physically moving about within acorresponding area of the guest environment 400. When reaching thelimits of the guest environment 400 (which can be defined by physical orvirtual barriers), the guest user 401 could use the control mechanism tovirtually teleport to a different location of the reconstructed hostenvironment 300′ and explore a new and otherwise inaccessible area ofthe reconstructed host environment 300′. The new area can correspond toa new limited predefined area of the reconstructed host environment 300′that comprises the location to which the guest user 401 chose toteleport. In some implementations, the new area can be centered aroundthe location to which the guest user 401 chose to teleport. As can beappreciated, such embodiments can enable the guest user 401 tophysically explore the entirety of the reconstructed host environment300′ despite physical limitations in the guest environment 400.

The digitally reconstructed host environment 300′ presented to the guestuser 401 can include virtual representations of environmental elements,such as physical objects that are physically present in the hostenvironment 300. More specifically, in the present embodiment, virtualreconstructions of the table 305′ and patient monitoring system 307′ areprovided in the reconstructed host environment 300′ at the same relativepositions of their physical counterparts 305, 307 in host environment300. As can be appreciated, the physical objects can be virtuallyreconstructed by the guest assembly 200 using any suitable process.Broadly described, the process can involve receiving, by the guestassembly 200, information relating to physical objects 305, 307 in thehost environment 300, and generating virtual 3D representations of saidobjects 305′, 307′ within the reconstructed host environment 300′. Insome embodiments, the guest assembly 200 can receive one or more 3Dmodels from host assembly 100, for example in the form of one or morepoint clouds or polygon meshes, and render such 3D models at theirsuitable positions in the virtual environment 300′. In otherembodiments, the guest assembly 200 can receive identifierscorresponding to one or more objects and their positions/orientations inthe host environment 300 from the host assembly 100. The guest assembly200 can subsequently use the received identifiers to retrievecorresponding 3D models from memory and/or from a database of predefinedmodels, and render the retrieved 3D models at the specified positionsand orientations within virtual environment 300′.

The process of generating 3D representations of physical objects canalso include representing surface textures of the physical objects. Inparticular, when rendering the 3D models of the physical objects 305′,307′ within the virtual environment 300′, textural elements such ascolors or patterns can be applied to the surfaces of the 3D models. Ascan be appreciated, textural elements can be rendered by guest assembly200 via any suitable process. Broadly described, the process can involvereceiving, by the guest assembly 200, information relating to surfacetextures of physical objects 305, 307 in the host environment 300, andapplying corresponding surface textures to rendered objects 305′, 307′within the reconstructed host environment 300′. In some embodiments, theguest assembly 200 can receive one or more surface images from hostassembly 100 and apply such images to the appropriate surfaces of therendered objects 305′, 307′. In some embodiments, the guest assembly 200can receive one or more indications of surface colors from host assembly100 and apply the surface colors to appropriate surfaces of the renderedobjects 305′, 307′. In yet further embodiments, the guest assembly 200can receive one or more identifiers corresponding to predefinedmaterials or patterns. The guest assembly 200 can subsequently use thereceived identifiers to retrieve textures corresponding to the materialsor patterns from memory and/or from a database, and apply the retrievedtextures to appropriate surfaces of the rendered objects 305′, 307′.

In an embodiment, the information relating to surface textures ofphysical objects 305, 307 can comprise images of the host environmentand perspective information corresponding to a position and orientationof a camera that captured the images. In such an embodiment, oncesurfaces are positioned within digitally reconstructed host environment300′, for example using received 3D polygonal meshes, the images of thehost environment can be applied to those surfaces as textures. Morespecifically, the guest assembly 200 can be configured to virtuallyproject the images from the host environment on the appropriate surfacesin reconstructed host environment 300′ using the perspectiveinformation. As an example, the guest assembly 200 can receive a videostream from host assembly 100 (for example via a peer-to-peer connectionbrokered via authorization server 500 as described above), the videostream having perspective information embedded therein. For each frameof the received video, the guest assembly 200 can extract the embeddedposition information to obtain the position and orientation within hostenvironment 300 of the camera that was used to capture the image, andposition a virtual projector within the reconstructed host environment300′ at the same position and orientation as the camera. The guestassembly 200 can subsequently project the image from the positionedvirtual projector onto the surfaces within the reconstructed hostenvironment 300′. This process can be repeated such that textures can beapplied from each frame of video received from the host assembly 200. Ascan be appreciated, this effectively transforms the camera of hostassembly 200 in host environment 300 into a virtual projector withinreconstructed host environment 300′. The virtual projector can moveabout the reconstructed environment 300′ as the camera of host assembly200 moves about in the host environment. In other words, the virtualprojector can follow the translation and rotation of the camera of hostassembly 200 in real time as the camera moves about in the hostenvironment 300.

In some instances, more detailed surface texture may be required tocommunicate relevant information relating to rendered objects. As anexample, patient monitoring system 307 can include a display 309 with agraphical user interface for visually representing detailed patientdata. To communicate such information effectively to guest user 401, thevirtual representation of the patient monitoring system 307′ can berendered with a detailed virtual representation of the display 309′. Inan embodiment, the guest assembly 200 can receive detailed images and/orcontinuous video of the display 309, and use such images and/or video tocreate a virtual representation of the display 309′ on the virtualpatient monitoring system 307′. In an embodiment, the guest assembly 200can receive a real-time rendering of the graphical user interface shownon the display 309, and apply said rendering to a surface of the patientmonitoring system 307′ as a virtual representation of the display 309′.In yet a further embodiment, the guest assembly 200 can receive datagathered by the patient monitoring system 307, and use such data torender a graphical user interface 309′ for display on the virtualpatient monitoring system 307′ that substantially corresponds to thegraphical user interface shown on the display 309 of the patientmonitoring system 307 in the host environment 300. As can beappreciated, the images, video, real-time rendering, or other data canbe received from host assembly 100 and/or from auxiliary device 109 overa peer-to-peer stream that can be brokered by authorization server 500as described above.

As can be appreciated, in addition to physical objects, the digitallyreconstructed host environment 300′ presented to the guest user 401 canalso include digital content. In the illustrated embodiment, the digitalcontent includes at least one virtual object, such as the shared virtualobject 310. The shared virtual object 310 can be presented to the guestuser 401 in the reconstructed host environment 300′ in the same relativeposition in 3D space as it would appear in the host environment 300. Asan example, if the host user 301 repositions the object 310 in the hostenvironment 300, the object 310 can be repositioned in real-time in thereconstructed host environment 300′ that is presented to guest user 401.Similarly, if the virtual object 310 is attached to a physical object,if the physical object is moved in the host environment, the virtualobject 310 can move in real-time to follow a virtual reconstruction ofthe physical object in the reconstructed host environment 300′. Asdescribed above, the virtual object 310 (whether shared or not) cancomprise a video stream received from auxiliary device 109. Live videoreceived from the auxiliary device 109 can thus be displayed as part thevirtual object 310 in the reconstructed environment 300′ in the same wayas in the host environment 300 as described above. The live video can bereceived by the guest assembly 200 directly from auxiliary device 109via a direct peer-to-peer connection and/or can be received via the hostdevice 100.

The digital content presented to guest user 401 can further include atleast one avatar 301′ corresponding to the host user 301, therebyallowing the guest user 401 to see and interact with a virtualrepresentation of the host user 301. The host avatar 301′ is controlledby the host user 301 and reflects the current position/perspective ofthe host user as the host user explores the 3D space of the hostenvironment 300. In some embodiments, the host avatar 301′ can reflectgestures that are made by the host user 301. As can be appreciated, inembodiments where a plurality of host users and host assemblies areprovided, the digital content can include a plurality of host avatarsrepresentative the position/perspective of each of the plurality of thehost users.

As can be appreciated, the above-described method and system can allowfor complex actions in 3D space to be more effectively and intuitivelycommunicated between users who are not present in the same physicalspace. In particular, a guest user and a host user can communicate andinteract from a distance within the physical environment of the hostuser. The guest user can be virtually transported into close physicalproximity of the host user and, with the help of their avatar, the guestuser can make gestures with their hands that can facilitatecommunication. The guest user is able to see the physical environment ofthe host user, thus allowing the guest user to observe and/or makereference to elements in the host user's environment. For example, usingtheir avatar, the guest user can point to a tool or other physicalobject in the host environment, and/or use gestures to explain to thehost user how to manipulate the tool while it is being held by the hostuser. As another example, the guest user can walk around the hostenvironment, observe and inspect objects from a first-personperspective, read information shown on physical displays in the hostenvironment, and communicate with the host user based on what theyobserved directly. If the guest user is a medical professional, this canallow the medical professional to virtually visit a patient's room,observe the patient, take readings from devices in the room, and provideinstructions to host user (such as a medical assistant) to providemedical care all the while not being physically present in the room. Ofcourse, many other applications are possible.

Although particular embodiments have been described above, it isappreciated that other variations are possible without departing fromthe scope of the disclosure. For example, although a single guest user401 was shown and described above, it is appreciated that a plurality ofguest users can be virtually transported into host environment 300simultaneously. Each guest user can have a virtual host environment 300′reconstructed in their guest respective environments. In someembodiments, two or more guest users can interact and be present in thesame guest environment.

As another example, although a single host user 301 was shown anddescribed, it is appreciated that a plurality of host users can bephysically present and interact in the host environment 300. Forexample, each host user can wear their own AR/MR device to interact withthe same guest users and digital content projected in the hostenvironment 300. In such embodiments, data gathered by the plurality ofAR/MR devices can be combined to gather more detailed information aboutthe physical environment and allow for a more accurate recreationthereof for guest users 401.

As another example, although the system and method were described inconnection with allowing a guest user to interact in a host environmentin real-time, it is appreciated that other configurations are possible.For example, the host environment and the host's actions can be recordedand played back at a later time. In such a configuration, the guest usercan be immersed in the environment of the host user to observe theenvironment and/or the host user's actions within the context of theenvironment at will. This can include playing back the recordingsmultiple times, at different speeds, and/or observing it from differentangles or perspectives in 3D space.

Finally, although in the above-described system and method only objectsin the host environment were reconstructed virtually, it is appreciatedthat other configurations are possible. For example, the guest assemblycan include an environment detection module that can allow detectingphysical objects in the guest environment for virtually reconstructingin the host environment. In some embodiments, only physical objects at aspecified location in the guest environment can be detected, such aswithin a predefined area, or within a predefined proximity to the guestuser. As an example, if the guest user is holding a physical tool, thattool can be scanned and virtually reconstructed in the host environmentsuch that the guest's avatar can be shown holding a virtualrepresentation of the tool.

In the foregoing specification, specific embodiments have beendescribed. However, one of ordinary skill in the art appreciates thatvarious modifications and changes can be made without departing from thescope of the present teachings. Accordingly, the specification andfigures are to be regarded in an illustrative rather than a restrictivesense, and all such modifications are intended to be included within thescope of present teachings.

The benefits, advantages, solutions to problems, and any element(s) thatmay cause any benefit, advantage, or solution to occur or become morepronounced are not to be construed as a critical, required, or essentialfeatures or elements of any or all the claims.

Moreover in this document, relational terms such as first and second andthe like may be used solely to distinguish one entity or action fromanother entity or action without necessarily requiring or implying anyactual such relationship or order between such entities or actions. Theterms “comprises,” “comprising,” “has”, “having,” “includes”,“including,” “contains”, “containing” or any other variation thereof,are intended to cover a non-exclusive inclusion, such that a process,method, article, or apparatus that comprises, has, includes, contains alist of elements does not include only those elements but may includeother elements not expressly listed or inherent to such process, method,article, or apparatus. An element proceeded by “comprises . . . a”, “has. . . a”, “includes . . . a”, “contains . . . a” does not, without moreconstraints, preclude the existence of additional identical elements inthe process, method, article, or apparatus that comprises, has,includes, contains the element. The terms “a” and “an” are defined asone or more unless explicitly stated otherwise herein. The terms“substantially”, “essentially”, “approximately”, “about” or any otherversion thereof, are defined as being close to as understood by one ofordinary skill in the art, and in one non-limiting embodiment the termis defined to be within 10%, in another embodiment within 5%, in anotherembodiment within 1% and in another embodiment within 0.5%. The term“coupled” as used herein is defined as connected, although notnecessarily directly and not necessarily mechanically. A device orstructure that is “configured” in a certain way is configured in atleast that way, but may also be configured in ways that are not listed.

It will be appreciated that some embodiments may be comprised of one ormore generic or specialized processors such as microprocessors, digitalsignal processors, customized processors and field programmable gatearrays (FPGAs) and unique stored program instructions (including bothsoftware and firmware) that control the one or more processors toimplement, in conjunction with certain non-processor circuits, some,most, or all of the functions of the method and/or apparatus describedherein. Alternatively, some or all functions could be implemented by astate machine that has no stored program instructions, or in one or moreapplication specific integrated circuits (ASICs), in which each functionor some combinations of certain of the functions are implemented ascustom logic. Of course, a combination of the two approaches could beused.

Moreover, an embodiment can be implemented as a computer-readablestorage medium having computer readable code stored thereon forprogramming a computer (e.g., comprising a processor) to perform amethod as described and claimed herein. Examples of suchcomputer-readable storage mediums include, but are not limited to, ahard disk, a CD-ROM, an optical storage device, a magnetic storagedevice, a ROM (Read Only Memory), a PROM (Programmable Read OnlyMemory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM(Electrically Erasable Programmable Read Only Memory) and a Flashmemory. Further, it is expected that one of ordinary skill,notwithstanding possibly significant effort and many design choicesmotivated by, for example, available time, current technology, andeconomic considerations, when guided by the concepts and principlesdisclosed herein will be readily capable of generating such softwareinstructions and programs and ICs with minimal experimentation.

1. A system for virtual interaction between a host user in a hostenvironment at a first physical location and a guest user at a secondphysical location, comprising: a guest assembly at the second physicallocation, the guest assembly comprising: a user interface moduleconfigured to: display a 3D virtual reconstruction of the hostenvironment at the second physical location based on receivedenvironmental information relating to the host environment; display avirtual avatar of the host user within the 3D virtual reconstruction ofthe host environment based on received interaction data relating to thehost user; and capture interaction data relating to the guest userwithin the virtual reconstruction of the host environment; and acommunications module configured to: receive the environmentalinformation relating to the host environment; receive the interactiondata relating to the host user in the host environment; and transmit theinteraction data relating to the guest user.
 2. The system according toclaim 1, further comprising a host assembly at the first physicallocation, the host assembly comprising: a user interface moduleconfigured to: display a virtual avatar of the guest user within thehost environment at the first physical location based on the interactiondata relating to the guest user; and capture the interaction datarelating to the host user in the host environment; an environmentdetection module configured to capture the environmental informationrelating to the host environment; and a communications module configuredto: receive the interaction data relating to the guest user from theguest assembly; transmit the interaction data relating to the host usercaptured by the user interface module; and transmit the environmentalinformation relating to the host environment captured by the environmentdetection module.
 3. The system according to claim 2, wherein theenvironmental information comprises volumetric data, further wherein theenvironment detection module comprises a volumetric sensor configured tocapture volumetric data corresponding to surfaces in the hostenvironment.
 4. The system according to claim 2, wherein theenvironmental information comprises texture data comprising images and aposition and orientation of a camera used to capture the images, furtherwherein the environment detection module comprises a camera configuredto capture images of the host environment, and at least one sensorconfigured to capture a position and orientation of the camera withinthe host environment.
 5. The system according to claim 1, furthercomprising an auxiliary device at the first physical location configuredto transmit a video stream, wherein the communications module of theguest assembly is configured to receive the video stream from theauxiliary device, and the user interface module of the guest assembly isconfigured to display a virtual object comprising the video stream inthe 3D virtual reconstruction of the host environment at the secondphysical location.
 6. The system according to claim 5, wherein theauxiliary device comprises a video input port configured for receiving avideo signal over a wired connection, and a wireless radio fortransmitting the video stream comprising the video signal over awireless network.
 7. The system according to claim 1, wherein the guestassembly comprises a camera for capturing video of the second physicallocation, further wherein the guest assembly is configured to determinea position of the guest user at the second physical location, to displaythe 3D virtual reconstruction of the host environment when the guestuser is determined to be within a predefined area at the second physicallocation, and to display the video of the second physical location whenthe guest user is determined to be outside of the predefined area.
 8. Amethod for virtual interaction between a host user at a first physicallocation and a guest user at a second physical location, the methodcomprising: receiving, via a guest assembly at the second physicallocation, environmental information relating to a host environment inthe first physical location; displaying, by the guest assembly, a 3Dvirtual reconstruction of the host environment at the second physicallocation using the received environmental information; receiving, viathe guest assembly, interaction data relating to the host user;displaying, by the guest assembly, a virtual avatar of the host userwithin the 3D virtual reconstruction of host environment in the secondphysical location using the received interaction data relating to thehost user; capturing, via the guest assembly, interaction data relatingto the guest user within the 3D virtual reconstruction of the hostenvironment; and transmitting, by the guest assembly to a host assemblyat the first physical location, the interaction data relating to theguest user, the interaction data allowing to display a virtual avatar ofthe guest user within the host environment at the first physicallocation.
 9. The method according to claim 8, further comprising:receiving, via the host assembly, the interaction data relating to theguest user; displaying, by the host assembly, the virtual avatar of theguest user within the host environment in the first physical locationusing the received interaction data relating to the guest user;capturing, via the host assembly, the interaction data relating to thehost user within the host environment; capturing, via the host assembly,the environmental information relating to the host environment; andtransmitting, by the host assembly to the guest assembly, theinteraction data relating to the host and the environmental informationrelating to the host environment.
 10. The method according to claim 8,wherein the environmental information comprises volumetric datacorresponding to surfaces in the host environment, and displaying the 3Dvirtual reconstruction of host environment comprises rendering virtualsurfaces using the volumetric data.
 11. The method according to claim10, wherein the volumetric data is captured by a volumetric sensorassociated with the host assembly.
 12. The method according to claim 10,wherein the environmental information comprises texture data, anddisplaying the 3D virtual reconstruction of host environment comprisesapplying the texture data to the virtual surfaces.
 13. The methodaccording to claim 12, wherein the texture data comprises images of thehost environment captured by a camera associated with the host assemblyand a position and orientation of the camera in the host environment,further wherein applying the texture data comprises virtually projectingthe images onto the virtual surfaces from a virtual projector atposition and orientation in the 3D virtual reconstruction of the hostenvironment corresponding to the position and orientation of the camerain the host environment.
 14. The method according to claim 8, furthercomprising: capturing, using a camera associated with the guestassembly, images of the second physical location; determining whetherthe guest user is within a predefined area of the second physicallocation; displaying the 3D virtual reconstruction of the hostenvironment if the guest user is determined to be within the predefinedarea; and displaying the images of the second physical location if theguest user is determine to be outside of the predefined area.
 15. Themethod according to claim 8, further comprising receiving, via the guestassembly, a video stream from an auxiliary device at the first physicallocation and displaying, by the guest assembly, a virtual objectcomprising the video stream in the 3D virtual reconstruction of the hostenvironment.
 16. The method according to claim 15, further comprisingreceiving, at the auxiliary device, a video signal over a wiredconnection and transmitting, by the video stream comprising the videosignal over a wireless radio.
 17. The method according to claim 15,further comprising establishing a peer-to-peer connection between theguest assembly and the auxiliary device for receiving the video stream.18. The method according to claim 17, wherein establishing thepeer-to-peer connection comprises: sending a request from the guestassembly to an authorization server to view the video stream;determining, by the authorization server, whether the guest user isauthorized to view the video stream; and if the guest user is authorizedto view the video stream, exchanging connection parameters with theguest assembly and the auxiliary device to enable the peer-to-peerconnection.
 19. The method according to claim 18, wherein thepeer-to-peer connection comprises end-to-end encryption, and theconnection parameters comprise encryption keys allowing for encryptionof the video stream by the auxiliary device, and decryption of the videostream by the guest assembly.
 20. A non-transitory computer-readablemedium for virtual interaction between a host user at a first physicallocation and a guest user at a second physical location, thecomputer-readable medium having instructions stored thereon which, whenexecuted, cause a processor to: receive, via a guest assembly at thesecond physical location, environmental information relating to a hostenvironment in the first physical location; display, by the guestassembly, a 3D virtual reconstruction of the host environment at thesecond physical location using the received environmental information;receive, via the guest assembly, interaction data relating to the hostuser; display, by the guest assembly, a virtual avatar of the host userwithin the 3D virtual reconstruction of host environment in the secondphysical location using the received interaction data relating to thehost user; capture, via the guest assembly, interaction data relating tothe guest user within the 3D virtual reconstruction of the hostenvironment; and transmit, by the guest assembly to a host assembly atthe first physical location, the interaction data relating to the guestuser, the interaction data allowing to display a virtual avatar of theguest user within the host environment at the first physical location.